MODEL SELECTION

Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP (2023.eacl-main)

Challenge:	Large Vision-Language Models (LVLMs) have transformed image captioning . existing evaluations lack standardized criteria and a standardized evaluation framework .
Approach:	They propose a leaderboard for evaluating detailed captions that addresses three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations.
Outcome:	The proposed model evaluates caption quality, descriptiveness, risks, and societal biases while tailoring criteria to user preferences.

Copied to clipboard

Xudong Han, Timothy Baldwin, Trevor Cohn

Responsible & Trustworthy NLP

Challenge:	Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct.
Approach:	They propose to clarify the current situation and plot a course for meaningful progress in fair learning by making clear inter-relations among the current gamut of methods and their relation to fairness theory.
Outcome:	The proposed approach addresses the practical problem of model selection, which involves a trade-off between fairness and accuracy and has led to systemic issues in fairness research.

Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances (2022.acl-short)

Copied to clipboard

Suvodip Dey, Ramamohan Kummara, Maunendra Desarkar

Dialogue Systems & Conversational Agents

Challenge:	Dialogue State Tracking (DST) is a task-oriented conversational agent that keeps track of key information exchanged during a conversation.
Approach:	They propose a new evaluation metric called Flexible Goal Accuracy to address shortcomings of JGA.
Outcome:	The proposed metric improves on existing metrics and improves performance of turn-level and non-cumulative belief state models.

Can Large Language Models Advance Crosswalks? The Case of Danish Occupation Codes (2025.naacl-srw)

Copied to clipboard

Bolei Ma, Cynthia A. Huang, Anna-Carolina Haensch

Don’t Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings (2020.emnlp-main)

Challenge:	Large Language Models (LLMs) are used to map classification systems to each other . however, their use is labor-intensive and requires domain expertise .
Approach:	They propose a prompt-based framework where LLMs perform similarity assessments between classification codes and identify final mappings through a guided decision process.
Outcome:	The proposed framework shows that LLMs perform better than the embedding-based framework in creating crosswalks.

Copied to clipboard

Phillip Keung, Yichao Lu, Julian Salazar, Vikas Bhardwaj

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data (2026.findings-acl)

Challenge:	Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning.
Approach:	They show that English dev accuracy makes it difficult to obtain reproducible results . they recommend providing oracle scores alongside zero-shot results if possible .
Outcome:	mBERT and XLM have shown strong performance on cross-lingual recognition, text classification, dependency parsing, and other tasks.

Copied to clipboard

Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, Leshem Choshen

Green & Sustainable NLP

Challenge:	Current evaluation practices, typically employing fixed-size benchmarks, are inherently wasteful, continuing to the predetermined sample size even when the CI reaches 2.5, saving 80% of the evaluation cost.
Approach:	They propose an adaptive evaluation framework that combines sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection and minimum detectable effect size.
Outcome:	The proposed framework reduces computational cost and reliability while maintaining statistical significance.

AutoML Meets Hugging Face: Domain-Aware Pretrained Model Selection for Text Classification (2025.naacl-srw)

Copied to clipboard

Parisa Safikhani, David Broneske

Information Extraction & Text Mining

Challenge:	Existing methods for fine-tuning text classification models are resource-intensive and require substantial computational power and time.
Approach:	They propose a corpus-driven domain mapping pipeline that integrates pre-fine-tuned models from Hugging Face Model Hub into AutoML systems to improve model selection.
Outcome:	The proposed pipeline improves model selection and streamlines workflows and reduces computational costs.

Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset (2022.emnlp-main)

Copied to clipboard

Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, Radu Soricut

Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia: Quality vs. Energy Consumption (2026.eacl-srw)

Challenge:	Existing studies on multilingual image captioning have been hampered by a lack of high-quality evaluation datasets.
Approach:	They present a dataset of 3600 images annotated with human-generated captions in 36 languages.
Outcome:	The proposed dataset shows that it is feasible to build multilingual image captioning models trained on machine-translated data.

Copied to clipboard

Patrycja Smits, Tomasz Walkowiak

AD-LLM: Benchmarking Large Language Models for Anomaly Detection (2025.findings-acl)

Challenge:	Retrieval-augmented generation systems are a dominant paradigm for knowledge-intensive applications.
Approach:	They evaluate language models from 4B to 70B parameters within a Polish Wikipedia-based RAG pipeline.
Outcome:	The proposed model selection process reduces energy consumption by 83% and improves quality.

Copied to clipboard

Tiankai Yang, Yi Nian, Li Li, Ruiyao Xu, Yuangang Li, Jiaqi Li, Zhuo Xiao, Xiyang Hu, Ryan A. Rossi, Kaize Ding, Xia Hu, Yue Zhao

Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation (2024.naacl-long)

Challenge:	Anomaly detection (AD) is an important machine learning task with many real-world uses, including fraud detection, medical diagnosis, and industrial monitoring.
Approach:	They propose a benchmark that evaluates how large language models (LLMs) can help with NLP anomaly detection.
Outcome:	The proposed model can perform zero-shot detection without tasks-specific training, data augmentation and model selection, and it can suggest unsupervised AD models.

Copied to clipboard

Se-eun Yoon, Zhankui He, Jessica Echterhoff, Julian McAuley

Dialogue Systems & Conversational Agents

Challenge:	Large language models show promise in simulating human-like behavior, raising the question of their ability to represent a diverse population of users.
Approach:	They propose a protocol to evaluate the degree to which language models can accurately emulate human behavior in conversational recommendation systems.
Outcome:	The proposed protocol evaluates five tasks to reveal deviations of language models from human behavior and offers insights on how to reduce deviations with model selection and prompting strategies.

Por Qué Não Utiliser Alla Språk? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer (2022.findings-naacl)

Copied to clipboard

Haoran Xu, Kenton Murray

Crossroads of Optimization under Uncertainty: How to Choose the Optimal Model (2026.findings-acl)

Challenge:	a lack of labeled data for low-resource languages leads to the need for effective cross-lingual transfer learning.
Approach:	They propose a mixed training method that trains on both source and target data with stochastic gradient surgery, a novel gradient-level optimization.
Outcome:	The proposed method outperforms current methods on all tasks and escapes overfitting issues.

Copied to clipboard

Chengxi She, Zhiqiang Chen, Xingyu Lu, Caihua Chen, Piaoyang Zhao, Xuedong Wang

Challenge:	Existing approaches to Optimization under Uncertainty (OuU) have inherent limitations and advantages.
Approach:	They propose a framework that automates the modeling and solving of six types of uncertainty models and generates mapping pairs to explore the potential relationship between optimization problems and optimal models.
Outcome:	The proposed framework achieves superior performance even on specific model types, with correlation analysis showing that data scale and specific scenario significantly influence model selection.

AutoAnoEval: Semantic-Aware Model Selection via Tree-Guided LLM Reasoning for Tabular Anomaly Detection (2026.findings-eacl)

Copied to clipboard

Suhee Yoon, Sanghyu Yoon, Ye Seul Sim, Seungdong Yoa, Dongmin Kim, Soonyoung Lee, Hankook Lee, Woohyung Lim

What Really Matters for Table LLMs? A Meta-Evaluation of Model and Data Effects (2026.findings-eacl)

Challenge:	Existing approaches to tabular anomaly detection fail to reflect domain specific nature of real-world anomalies.
Approach:	They propose a framework that constructs pseudo-evaluation sets with semantically grounded synthetic anomalies.
Outcome:	The proposed framework generates pseudo-evaluation sets with semantically grounded synthetic anomalies.

Copied to clipboard

Naihao Deng, Sheng Zhang, Henghui Zhu, Shuaichen Chang, Jiani Zhang, Alexander Hanbo Li, Chung-Wei Hang, Hideo Kobayashi, Yiqun Hu, Patrick Ng

Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction (P19-1)

Challenge:	a series of paradigm shifts have come with distinct characteristics and challenges associated with table modeling.
Approach:	They propose to replicate four table LLMs by instruction-tuning three foundation models on four existing datasets.
Outcome:	The results show that base model choice plays a more dominant role than training data itself.

Copied to clipboard

Lifeng Jin, William Schuler

Text Error Correction

Challenge:	In unsupervised grammar induction, data likelihood is only weakly correlated with parsing accuracy, especially at convergence after multiple runs.
Approach:	They propose to use VAS instead of data likelihood to find better grammars by examining linguistically-motivated constraints related to syntax.
Outcome:	The proposed model is better suited for word order typology classification than data likelihood.

Empirical Study of Zero-shot Keyphrase Extraction with Large Language Models (2025.coling-main)

Copied to clipboard

Byungha Kang, Youhyun Shin

Using J-K-fold Cross Validation To Reduce Variance When Tuning NLP Models (C18-1)

Challenge:	a prompting-based approach can effectively supersede traditional KE methods, a study shows . our code is available at https://github.com/kangnlp/zero-shot-keyphrase-extraction-with-LLMs.
Approach:	They propose four prompting strategies for zero-shot keyphrase extraction using Large Language Models.
Outcome:	The proposed prompting strategies outperform state-of-the-art prompting methods on KE benchmark datasets.

Copied to clipboard

Henry Moss, David Leslie, Paul Rayson

Text Classification

Challenge:	a recent study shows that performance estimations are unstable and variable . this makes it difficult to use parameter tuning and model selection .
Approach:	They propose to use a less variable CV method to evaluate performance . they propose lower choices of K than are typically seen in the NLP literature .
Outcome:	The proposed method can be used for parameter tuning and performance estimation, but it is unstable and unstable.

Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputs (2025.coling-main)

Copied to clipboard

Junhao Chen, Xiang Li, Xiaojun Ye, Chao Li, Zhaoxin Fan, Hao Zhao

FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms (P19-1)

Challenge:	Existing 3D AIGC methods don’t fully unleash human creativity.
Approach:	They propose a framework that generates 3D content from multimodal inputs . they propose 198 multimodal text inputs for 3D generation tasks .
Outcome:	The proposed framework generates 3D content from multimodal inputs without human intervention.

Copied to clipboard

Henry Moss, Andrew Moore, David Leslie, Paul Rayson

Sentiment Analysis

Challenge:	evaluating models is a non-trivial task and requires extensive data and data splits to produce reliable comparisons.
Approach:	They propose a model selection approach that reduces the computational resources required to compare models based on single choices of random seeds.
Outcome:	The proposed model selection approach reduces the computational resources required to identify state-of-the-art models from large datasets.

To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages (2024.naacl-long)

Copied to clipboard

Benedikt Ebing, Goran Glavaš

Multilinguality

Challenge:	XLT with multilingual language models is superfluous, says a new study . mBERT, XLM-R and mT5 are effective for cross-lingual transfer, authors say .
Approach:	They propose to use multilingual language models to improve cross-lingual transfer (XLT) they propose to add reliable translations to training data for XLT even for non-MT languages .
Outcome:	The proposed approaches outperform zero-shot XLT with mLMs, the authors show . the authors believe their findings warrant a broader inclusion of more robust translation-based baselines in XL research.

Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging (2023.acl-long)

Copied to clipboard

Fabian David Schmidt, Ivan Vulić, Goran Glavaš

Robustness in NLP

Challenge:	Massively multilingual language models have shown strong performance in zero-shot (ZS-XLT) and few-shot cross-lingual transfer setups where models are fine-tuned on task data in a source language are transferred without any or with only a few annotated instances to the target language(s).
Approach:	They propose a method that averages different checkpoints during task fine-tuning to improve model robustness.
Outcome:	The proposed method overestimates model performance in cross-lingual transfer setups where models are evaluated at checkpoints that generalize best to validation instances in the target languages.

Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again (2022.findings-emnlp)

Copied to clipboard

Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, Yu Su

Low-Resource NLP

Challenge:	Large pre-trained language models (PLMs) such as GPT-3 have shown strong in-context learning capabilities, which are appealing for domains such as biomedicine that feature high and diverse demands of language technologies but also high data annotation costs.
Approach:	They propose to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs on two representative biomedical information extraction tasks: named entity recognition and relation extraction.
Outcome:	The proposed model underperforms on two representative biomedical information extraction tasks.

Firewall Routing: Blocking Leads to Better Hybrid Inference for LLMs (2025.emnlp-main)

Copied to clipboard

Runyu Peng, Yunhua Zhou, Kai Lv, Yang Gao, Qipeng Guo, Xipeng Qiu

Challenge:	Large language models have significantly enhanced performance across various NLP tasks . high computational costs and latency associated with deploying such models pose bottlenecks .
Approach:	They propose a dynamic hybrid inference framework that efficiently selects between a strong and a weak LLM based on the complexity of the query.
Outcome:	The proposed method outperforms existing routing strategies by up to 5.29% in APGR . large models often introduce higher latency, making them less suitable for real-time or resource-constrained applications.

Arena-lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons (2025.emnlp-main)

Copied to clipboard

Seonil Son, Ju-Min Oh, Heegon Jin, Cheolhun Jang, Jeongbeom Jeong, KunTae Kim

We Need to Talk About train-dev-test Splits (2021.emnlp-main)

Challenge:	Current benchmarks typically compare system outputs against baselines, but this method yields lower reliability than direct comparison.
Approach:	They propose to integrate tournament structure on top of head-to-head comparison.
Outcome:	The proposed model achieves higher reliability with fewer comparisons even with smaller datasets or weaker judges.

Copied to clipboard

Rob van der Goot

Challenge:	Standard train-dev-test splits used to benchmark multiple models are now used in NLP . comparing multiple versions of the same model on the test data leads to overfitting and "expiration" of test sets.
Approach:	They propose to use a tune-set when developing neural network methods to do model picking.
Outcome:	The proposed model picker is more robust against the evaluated hyperparameter ranges than the standard split split.

XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML (2025.emnlp-main)

Copied to clipboard

Ernesto Luis Estevanell Valladares, Suilan Estevez-Velarde, Yoan Gutierrez, Andrés Montoyo, Ruslan Mitkov

Responsible & Trustworthy NLP

Challenge:	XAutoLM is a meta-learning-augmented framework that can be used to optimize discriminative and generative LM fine-tuning pipelines.
Approach:	They propose a meta-learning-augmented AutoML framework that reuses past experiences to optimize discriminative and generative LM fine-tuning pipelines efficiently.
Outcome:	XAutoLM surpasses zero-shot optimizer’s peak F1 on five of six tasks, reduces mean evaluation time of pipelines by up to 4.5x, and uncovers 50% more pipelines above zero- shot Pareto front.

Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models (2023.emnlp-main)

Copied to clipboard

Di Wu, Wasi Ahmad, Kai-Wei Chang

Climbing the Tower of Treebanks: Improving Low-Resource Dependency Parsing via Hierarchical Source Selection (2021.findings-acl)

Challenge:	Keyphrase generation is a longstanding task in NLP with widespread applications.
Approach:	They propose a likelihood-based decode-select algorithm for seq2seq PLMs that improves greedy search by an average of 4.7% semantic F1 across five datasets.
Outcome:	The proposed algorithm improves greedy search by an average of 4.7% semantic F1 across five datasets.

Copied to clipboard

Goran Glavaš, Ivan Vulić

Syntactic Text Processing

Challenge:	Recent work on multilingual dependency parsing focused on developing highly multilingual parsers . a recent major paradigm shift in NLP towards largescale pretrained language models has reduced the downstream relevance of supervised syntactic parse.
Approach:	They propose a heuristic approach to multilingual dependency parsing that heurs out the "one model to rule them all" approach by hierarchically clustering all Universal Dependencies languages based on their syntactic similarity .
Outcome:	The proposed approach outperforms a "one model to rule them all" approach with a heuristic selection of languages and treebanks for a target language.

UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2022.acl-long)

Copied to clipboard

Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, Madian Khabsa

Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference (2022.naacl-main)

Challenge:	Existing methods for parameter-efficient language model tuning (PELT) match the performance of fine-tuning with fewer trainable parameters.
Approach:	They propose a framework which integrates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism.
Outcome:	The proposed framework outperforms fine-tuning methods on the GLUE benchmark and achieves 14% gains over the best individual PELT method.

Copied to clipboard

Emīls Kadiķis, Vaibhav Srivastav, Roman Klinger

Reasoning

Challenge:	a method for learning an NLI model is time-consuming and resource-intensive, but it can save time and resources.
Approach:	They propose a method for predicting model performance without fine-tuning it . they compare sentence embeddings with cosine similarity to classifiers .
Outcome:	The proposed method can save time and resources by comparing pre-trained models to real-world datasets.

Model Selection for Cross-lingual Transfer (2021.emnlp-main)

Copied to clipboard

Yang Chen, Alan Ritter

Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments (2022.findings-emnlp)

Challenge:	Existing work has relied on English dev data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices.
Approach:	They propose a machine learning approach that uses the fine-tuned model’s internal representations to predict its cross-lingual capabilities.
Outcome:	The proposed model selects better than English validation data across twenty five languages, including eight low-resource languages, and often achieves comparable results to model selection using target language development data.

Copied to clipboard

Maor Ivgi, Yair Carmon, Jonathan Berant

F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering (2020.emnlp-main)

Challenge:	Neural scaling laws define a predictable relationship between a model’s parameter count and its performance after training in the form of a power law.
Approach:	They perform an empirical investigation of language understanding tasks and evaluate their results to determine whether scaling laws can be used to accelerate model development.
Outcome:	The proposed scaling laws can be exploited for debugging convergence when training large models, and can predict the performance of larger models.

Copied to clipboard

Hendrik Schuff, Heike Adel, Ngoc Thang Vu

Explainability & Interpretability in NLP

Challenge:	Existing models and evaluation settings have shortcomings regarding the coupling of answer and explanation which might cause serious issues in user experience.
Approach:	They propose a hierarchical model and a new regularization term to strengthen the coupling of answer and explanation and two evaluation scores to quantify the couple.
Outcome:	The proposed model strengthens the answer-explanation coupling and provides evaluation scores that align with user experience.

Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models (2023.findings-emnlp)

Copied to clipboard

Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, Heng Ji

Responsible & Trustworthy NLP

Challenge:	Large Language Models (LLMs) have been experiencing seismic growth in size and capabilities, radically transforming the field of NLP.
Approach:	They propose a generalized variant of iterative self-critique and self-refinement devoid of external influence and a ranking metric to find the optimal model for a given task considering refined performance and cost.
Outcome:	The proposed model improves 8.2% from baseline and even with extremely small memory footprints, outperforms ChatGPT post-refinement.

On the Role of Supervision in Unsupervised Constituency Parsing (2020.emnlp-main)

Copied to clipboard

Haoyue Shi, Karen Livescu, Kevin Gimpel

Low-Resource NLP

Challenge:	Recent work on unsupervised constituency parsing uses labeled examples for tuning . a few-shot parser with labeles can outperform other approaches by a significant margin .
Approach:	They propose to use as few labeled examples as possible for model development . they propose to train existing models on the same labeles they access .
Outcome:	The proposed model outperforms other models on the WSJ development set by a significant margin . the proposed model can be further improved by augmentation and self-training .

From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinement (2025.emnlp-main)

Copied to clipboard

JianZhi Yan, Le Liu, Youcheng Pan, Shiwei Chen, Zike Yuan, Yang Xiang, Buzhou Tang

Challenge:	Chain-of-Thought reasoning introduces significant inference latency due to its verbosity.
Approach:	They propose a framework that leverages token elasticity phenomenon to progressively compress CoTs via multiround refinement.
Outcome:	The proposed method achieves an average accuracy improvement of 5.6% over state-of-the-art baselines while reducing CoT length by an average of 47 tokens and significantly lowering latency.

Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking (2024.emnlp-main)

Copied to clipboard

Jun Bai, Zhuofan Chen, Zhenzi Li, Hanhua Hong, Jianfei Zhang, Chen Li, Chenghua Lin, Wenge Rong

Information Extraction & Text Mining

Challenge:	Existing methods for text ranking are based on intuition, but their estimated transferability may not align well with the objectives of text ranking.
Approach:	They propose to compute expected rank as transferability, explicitly reflecting the model’s ranking capability.
Outcome:	The proposed method shows significant improvements over previous classification-oriented TE methods, human intuition, and ChatGPT with minor time consumption.

SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone (2025.findings-emnlp)

Copied to clipboard

Nishant Subramani, Alfredo Gomez, Mona T. Diab

Paraphrasing

Challenge:	Modern language models are evaluated on large benchmarks, which are difficult to make sense of.
Approach:	They propose a framework to Simplify Benchmark Analysis using model-centric evaluation numbers.
Outcome:	The proposed framework can be applied to HELM, MMLU, and BigBenchLite benchmarks.

An HMM Approach with Inherent Model Selection for Sign Language and Gesture Recognition (2020.lrec-1)

Copied to clipboard

Sandrine Tornay, Oya Aran, Mathew Magimai Doss

Challenge:	despite the extensive use of HMMs for sign recognition, determining the HMM structure remains a challenge . despite their success in modeling sequential and multivariate data, establishing the structure remains challenging .
Approach:	They propose a continuous HMM framework for modeling and recognizing isolated signs . they propose to optimize the number of states for each sign separately during recognition .
Outcome:	The proposed model performs better on three different datasets and is competitive with existing models.

When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024.acl-long)

Copied to clipboard

Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, Haidar Khan

On Leakage of Code Generation Evaluation Datasets (2024.findings-emnlp)

Challenge:	Existing leaderboards are often taken at face value, but this is costly . a recent study shows that minor perturbations to the benchmark result in rankings up to 8 positions.
Approach:	They propose to use a hybrid scoring method for answer selection for large language models . they find that minor perturbations to the benchmark result in rankings changes .
Outcome:	The proposed model is a hybrid scoring method, the authors argue . the proposed model could be used to improve the performance of large language models .

Copied to clipboard

Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, Matthias Gallé

Programming Languages in NLP

Challenge:	In this paper, we discuss contamination by code generation test sets in large language models.
Approach:	They propose to use Python to test code generation test sets for contamination . they find that code generation is an important skill for large language models to master .
Outcome:	The proposed benchmarks are uncontaminated and provide a new insight into code generation.

One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging for Cross-Lingual Transfer (2023.findings-emnlp)

Copied to clipboard

Fabian David Schmidt, Ivan Vulić, Goran Glavaš

Cross-Lingual Transfer

Challenge:	Existing methods for zero-shot cross-lingual transfer are unreliable due to the lack of pretraining data.
Approach:	They propose to accumulatively average model snapshots from different runs into a single model.
Outcome:	The proposed protocol decouples performance maximization from hyperparameter tuning.

Choosing a Model, Shaping a Future: Comparing LLM Perspectives on Sustainability and its Relationship with AI (2025.findings-emnlp)

Copied to clipboard

Annika Bush, Meltem Aksoy, Markus Pauly, Greta Ontrup

AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science (2025.findings-emnlp)

Challenge:	Claude, DeepSeek, GPT, LLaMA, and Mistral conceptualize sustainability and its relationship with AI.
Approach:	They investigated how five state-of-the-art LLMs conceptualize sustainability and its relationship with AI.
Outcome:	The proposed models differed significantly between models, revealing significant biases and biase . Claude, DeepSeek, GPT, LLaMA, and Mistral conceptualize sustainability and its relationship with AI.

Copied to clipboard

An Luo, Xun Xian, Jin Du, Fangqiao Tian, Ganghua Wang, Ming Zhong, Shengchun Zhao, Xuan Bi, Zirui Liu, Jiawei Zhou, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, Jie Ding

Robustness in NLP

Challenge:	Large language models (LLMs) have advanced the automation of data science workflows, yet it remains unclear whether they can critically leverage external domain knowledge as human data scientists do in practice.
Approach:	They propose a benchmark to evaluate how large language models handle external domain knowledge in tabular prediction tasks.
Outcome:	The proposed model evaluates whether it can critically leverage external domain knowledge as human data scientists do in practice.

Graph vs. Sequence: An Empirical Study on Knowledge Forms for Knowledge-Grounded Dialogue (2023.emnlp-main)

Copied to clipboard

Yizhe Yang, Heyan Huang, Yuhang Liu, Yang Gao

Natural Language Interfaces

Challenge:	Knowledge-grounded dialogue systems can generate informative responses based on dialogue history and external knowledge source.
Approach:	They conduct a thorough experiment to determine the optimal knowledge form, mutual effects between knowl- edge and model selection, and the few-shot performance of knowledge.
Outcome:	The proposed method combines knowledge-grounded dialogue with human-generated dialogues to generate informative and meaningful responses.

SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science (2026.acl-long)

Copied to clipboard

Wonduk Seo, Juhyeon Lee, Yanjun Shao, Qingshan Zhou, Seunghyun Lee, Yi Bu